Dynamic Load Balancing in Stream Processing Pipelines Containing Stream-Static Joins

Data stream processing systems are used to continuously run mission-critical applications for real-time monitoring and alerting. These require high throughput low latency process incoming data streams in real time. However, changes the distribution of over time can cause partition skew, which is defined as an unequal partitions among workers, resulting sub-optimal due unbalanced load. This paper presents first solution designed specifically address skew context joining streaming static data. Our uses state-of-the-art principles monitor load, detect load imbalance, dynamically redistribute partitions, achieve optimal balance. To accomplish this, our leverages collocation data, while considering join subsequent operations. Finally, we present results experimental evaluation, compared four pipelines containing such a join. The show that achieved significantly higher lower than competing approaches.